BPF is your Friend

CapLoader comes with support for Berkeley Packet Filter (BPF), which makes it possible to filter network traffic based on IP addresses, protocols and port numbers without using external tools. Being able to filter captured network traffic is crucial when analyzing large sets of PCAP files as well as in order to hunt down compromised hosts with Rinse Repeat Intrusion Detection.

There are two ways to apply filters with BPF in CapLoader; you can either apply an input filter before loading your PCAPs, or you can apply a display filter after the capture files have been loaded.

Input Filter

The fastest way to filter a large set of PCAP files with CapLoader is to enter an Input Filter before loading the capture files. Having an input filter will speed up the loading time significantly, since CapLoader will not need to analyze packets and flows that don't match the BPF syntax. The downside is that you will need to know beforehand what filter you want to use. In order to apply a changed input filter you need to reload the loaded PCAP files (pressing F5 will do this for you).

Display Filter

CapLoader supports display filters in order to allow filters to be changed on the fly, without having to reload the PCAP files. As the name implies, display filters affect what flows/services/hosts that are displayed in CapLoader. A changed display filter does not require the dataset to be reloaded, but it does require the GUI to update the visible flows. This GUI update will be somewhat slower compared to when setting an input filter.

BPF Syntax

CapLoader's BPF implementation does not support the full BPF syntax. In fact, only the most central primitives are implemented, which are:

More complex filter expressions can be built up by using the words and, or, not and parentheses to combine primitives. Here are some examples:

• host 8.8.8.8 and udp port 53
• net 199.16.156.0/22 and port 80
• (port 80 or port 443) and not host 192.168.0.1

For all boolean algebra geeks out there we can confirm that our BPF implementation gives and precedence over or, which means that the last example above would give a different result if the parentheses were removed.

Keeping it Short

Steve McCanne gave a keynote presentation at SharkFest 2011, where he talked about how he created BPF. Steve's work was guided by Van Jacobson, who challenged him to make the BPF syntax human friendly rather than requiring the user to type a clunky filtering syntax. We've adopted this thinking and therefore allow filters like these:

• 10.1.1.3
  Flows to or from IP address 10.1.1.3. Translates to “ip host 10.1.1.3”


• 128.3/16
  Flows to or from the 128.3.0.0/16 network. Translates to “ip net 128.3.0.0/16”


• port 53
  Flows to or from TCP, UDP or SCTP port 53.

Try it for Free!

We've made the BPF implementation available even in the free version of CapLoader. You don't need to register to get the free version; just download, extract and run. The tool is portable, so you won't even have to install it. Visit https://www.netresec.com/?page=CapLoader to grab a copy and start filtering!

UPDATE 2016-05-23

With the release of CapLoader 1.4 it is now possible to apply Display Filters not only to the Flows tab, but also to the Services and Hosts tab.

Posted by Erik Hjelmvik on Monday, 30 November 2015 08:15:00 (UTC/GMT)

Tags: #CapLoader​ #BPF​ #PCAP​ #Berkeley Packet Filter​ #filter​ #IP​ #port​ #CIDR​

CapLoader 1.3 Released

A new version of our heavy-duty PCAP parser tool CapLoader is now available. There are many new features and improvements in this release, such as the ability to filter flows with BPF, domain name extraction via passive DNS parser and matching of domain names against a local white list.

Filtering with BPF

The main focus in the work behind CapLoader 1.3 has been to fully support the Rinse-Repeat Intrusion Detection methodology. We've done this by improving the filtering capabilities in CapLoader. For starters, we've added an input filter, which can be used to specify IP addresses, IP networks, protocols or port numbers to be parsed or ignored. The input filter uses the Berkeley Packet Filter (BPF) syntax, and is designed to run really fast. So if you wanna analyze only HTTP traffic you can simply write “port 80” as your input filter to have CapLoader only parse and display flows going to or from port 80. We have also added a display filter, which unlike Wireshark also uses BPF. Thus, once a set of flows is loaded one can easily apply different display filters, like “host 194.9.94.80” or “net 192.168.1.0/24”, to apply different views on the parsed data.

The main differences between the input filter and display filter are:

• Input filter is much faster than the display filter, so if you know beforehand what ports, protocols or IP addresses you are interested in then make sure to apply them as an input filter. You will notice a delay when applying a display filter to a view of 10.000 flows or more.
• In order to apply a new input filter CapLoader has to reload all the opened PCAP files (which is done by pressing F5). Modifying display filters, on the other hand, only requires you to press Enter or hit the “Apply” button.
• Previously applied display filters are accessible in a drop-down menu in the GUI, but no history is kept of previous input filters.

NetFlow + DNS == true

The “Flows” view in CapLoader gives a great overview of all TCP, UDP and SCTP flows in the loaded PCAP files. However, it is usually not obvious to an analyst what every IP address is used for. We have therefore added a DNS parser to CapLoader, so that all DNS packets can be parsed in order to map IP addresses to domain names. The extracted domain names are displayed for each flow, which is very useful when performing Rinse-Repeat analysis in order to quickly remove “known good servers” from the analysis.

Leveraging the Alexa top 1M list

As we've show in in our previous blog post “DNS whitelisting in NetworkMiner”, using a list of popular domain names as a whitelist can be an effective method for finding malware. We often use this approach in order to quickly remove lots of known good servers when doing Rinse-Repeat analysis in large datasets.

Therefore, just as we did for NetworkMiner 1.5, CapLoader now includes Alexa's list of the 1 million most popular domain names on the Internet. All domain names, parsed from DNS traffic, are checked against the Alexa list. Domains listed in the whitelist are shown in CapLoader's “Server_Alexa_Domian” column. This makes it very easy to sort on this column in order to remove (hide) all flows going to “normal” servers on the Internet. After removing all those flows, what you're left with is pretty much just:

• Local traffic (not sent over the Internet)
• Outgoing traffic to either new or obscure domains

Manually going through the remaining flows can be very rewarding, as it can reveal C2 traffic from malware that has not yet been detected by traditional security products like anti-virus or IDS.

Many new features in CapLoader 1.3

The new features highlighted above are far from the only additions made to CapLoader 1.3. Here is a more complete list of improvements in this release:

• Support for “Select Flows in PCAP” to extract and select 5-tuples from a PCAP-file. This can be a Snort PCAP with packets that have triggered IDS signatures. This way you can easily extract the whole TCP or UDP flow for each signature match, instead of just trying to make sense of one single packet per alert.
• Improved packet carver functionality to better carve IP, TCP and UPD packets from any file. This includes memory dumps as well as proprietary and obscure packet capture formats.
• Support for SCTP flows.
• DNS parser.
• Alexa top 1M matching.
• Input filter and display filter with BPF syntax.
• Flow Producer-Consumer-Ratio PCR.
• Flow Transcript can be opened simply by double-clicking a flow.
• Find form updated with option to hide non-matching flows instead of just selecting the flows that matched the keyword search criteria.
• New flow transcript encoding with IP TTL, TCP flags and sequence numbers to support analysis of Man-on-the-Side attacks.
• Faster loading of previously opened files, MD5 hashes don't need to be recalculated.
• A selected set of flows in the GUI can be inverted simply by right-clicking the flow list and selecting “Invert Selection” or by hitting Ctrl+I.

Downloading CapLoader 1.3

All these new features, except for the Alexa lookup of domain names, are available in our free trial version of CapLoader. So to try out these new features in CapLoader, simply grab a trial download here:
https://www.netresec.com/?page=CapLoader#trial (no registration needed)

All paying customers with an older version of CapLoader can grab a free update for version 1.3 at our customer portal.

Posted by Erik Hjelmvik on Monday, 28 September 2015 07:30:00 (UTC/GMT)

Tags: #CapLoader​ #BPF​ #Berkeley Packet Filter​ #Rinse-Repeat​ #DNS​ #Alexa​ #PCAP​ #Passive DNS​ #NetFlow​ #Malware​ #C2​